Systems and methods for modeling and processing functional magnetic resonance image data using full-brain vector auto-regressive model

ABSTRACT

Systems and methods for modeling functional magnetic resonance image datasets using a multivariate auto-regressive model which captures temporal dynamics in the data, and creates a reduced representation of the dataset representative of functional connectivity of voxels with respect to brain activity. Raw spatio-temporal data is processed using a multivariate auto-regressive model, wherein coefficients in the model with high weights are retained as indices that best describe the full spatio-temporal data. When there are a relatively small number of temporal samples of the data, sparse regression techniques are used to build the model. The model coefficients are used to perform data processing functions such as indexing, prediction, and classification.

FIELD OF THE INVENTION

The present invention relates generally to systems and methods formodeling and processing functional magnetic resonance image datasetsand, more specifically, systems and methods for modeling functionalmagnetic resonance image datasets using a multivariate auto-regressivemodel which captures temporal dynamics in the data, and creates areduced representation of the dataset representative of functionalconnectivity of voxels with respect to brain activity.

BACKGROUND

Recent advances in medical imaging technology have introduced functionalmagnetic resonance imaging (fMRI) capable of acquiring sequences ofimages of brain activity by measuring changes in blood oxygenationlevels. Functional magnetic resonance imaging is increasingly used inthe medical field to scan subjects, both normal and diseased. The fMRIdata is a 4-dimensional dataset involving 3 spatial dimensions and onetemporal dimension. An fMRI dataset is very large and difficult tovisualize for making meaningful conclusions. Typically, this data isprocessed by different analysis techniques to generate a number of“human viewable maps” which are then used to study the fMRI data andreach conclusions.

One of the most common approaches to processing fMRI data is known asthe general linear model (GLM) technique, which makes use of anexperimental protocol while a subject is being scanned. The GLMtechnique produces spatial maps of brain activity, which indicate thoseareas of the brain that are active for a given experiment (stimulus)being conducted on the target subject. More specifically, with the GLMtechnique, activity in different regions of while the subject is notconducting the given experimental task. A thresholding protocol orlinear analysis is then performed on each dataset to determine ifcertain measured activity is beyond noise. Once spatial maps of brainactivity are obtained for the measured activity with and without theexperimental stimulus, the spatial maps are compared to determine whichareas of the brain are differentially activated.

With the GLM protocol, the resulting spatial maps only provideinformation based on measured activity of different regions of thebrain, independent of each other, and do not provide any informationregarding how activity in one voxel relates to, or affects, or otherwisetriggers, activity of another voxel. In other words, the spatial mapsderived using the GLM technique do not provide any information about thedynamics of ongoing brain activity which is not directly related to theexperimental task conducted. Therefore, it is not possible to fullysummarize the fMRI data using such techniques and later use it forindexing, prediction, or classification purposes.

SUMMARY OF THE INVENTION

Aspects of the present invention generally include systems and methodsfor modeling functional magnetic resonance image datasets using amultivariate auto-regressive model that captures temporal dynamics inthe data, and creates a reduced representation of the datasetrepresentative of functional connectivity of voxels with respect tobrain activity. More specifically, aspects of the invention includesystems and method that fit raw spatio-temporal data with a multivariateauto-regressive model. This model captures the temporal dynamics in thedata, and creates a reduced representation of the data. Coefficients inthe model with high weights can be retained as indices that bestdescribe the full spatio-temporal data. When there are a relativelysmall number of temporal samples of the data, sparse regressiontechniques are used to build the model. The model coefficients are usedto perform data processing functions such as indexing, prediction, andclassification.

More specifically, in one aspect of the invention, a method to performan image data processing operation includes obtaining a rawspatio-temporal dataset acquired from scanning a brain of a subjectperforming a given task, constructing a full spatio-temporal model usingthe raw spatio-temporal dataset, wherein the full spatio-temporal modelrepresents brain activity that occurs in all regions of the subject'sbrain in response to the subject performing the given task, selectingmodel parameters from the full spatio-temporal model which meet orexceed a threshold parameter that defines a level of causal relationbetween voxels in the acquired dataset, generating a reduced modelrepresentation of the full spatio-temporal model using the selectedmodel parameters, generating a vector representing the reduced model,and using the vector to perform an image data processing operation.

In another aspect of the invention, a method for predicting future brainactivity includes obtaining a previously generated spatio-temporal modelrepresenting brain activity of a subject, which model was previouslygenerated from scan data collected with the subject performing a giventask, initializing the previously generated spatio-temporal model withcurrent brain activity data derived from a scan of the subject's brainwhile the subject is performing the same given task, and predictingfuture brain activity of the subject using the previously generatedspatio-temporal model based on the current brain activity datainitializing said model. The biofeedback may be provided to the subjectbased on the predicted future brain activity to provide an indication tothe subject of the predicted future brain activity of the subject.

In accordance with another aspect of the invention, an apparatus formodeling and processing image data is provided. An apparatus includes amemory and a processor coupled to the memory, wherein the apparatus isoperative to perform methods for modeling and processing image data asdescribed above.

In accordance with yet another aspect of the invention, a computerprogram product to perform an image data processing operation isprovided. The computer program product comprises a computer readablestorage medium having computer readable program code embodied therewith.The computer readable program code comprises computer readable programcode configured to perform methods for modeling and processing imagedata as described above.

These and other aspects, and features of the present invention willbecome apparent from the following detailed description of illustrativeembodiments thereof, which is to be read in connection with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a high-level block diagram of a system for modeling andprocessing functional magnetic resonance imaging data according to anaspect of the invention.

FIG. 2 is a flow diagram of a method for modeling and processingfunctional magnetic resonance imaging data according to an aspect of theinvention.

FIG. 3 depicts a directed node graph representing functionalconnectivity and causality of one voxel on another voxel, which isderived from modeling functional magnetic resonance image data,according to an aspect of the invention.

FIG. 4 is a flow diagram of a method for generating a reducedrepresentation of a full spatio-temporal model of brain activity of asubject, according to an aspect of the invention.

FIG. 5 is a flow diagram of a method for generating a vectorrepresentation of a reduced spatio-temporal model of brain activity of asubject for use in database searching and classification, according toan aspect of the invention.

FIG. 6 is a flow diagram of a method for performing a database searchusing a vector representation of a reduced spatio-temporal model ofbrain activity of a subject, according to an aspect of the invention.

FIG. 7 is a flow diagram of a method for performing classification usinga vector representation of a reduced spatio-temporal model of brainactivity of a subject, according to an aspect of the invention.

FIG. 8 is a flow diagram of a method for predicting brain activity usinga reduced representation of a full spatio-temporal model of brainactivity of a subject, according to an aspect of the invention.

FIG. 9 depicts a computer system that may be useful in implementing oneor more aspects and/or elements of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Aspects of the present invention generally include systems and methodsfor modeling functional magnetic resonance image datasets using amultivariate auto-regressive model that captures temporal dynamics inthe data, and creates a reduced representation of the datasetrepresentative of functional connectivity of voxels with respect tobrain activity. More specifically, aspects of the invention includesystems and method that fit raw spatio-temporal data with a multivariateauto-regressive model. This model captures the temporal dynamics in thedata, and creates a reduced representation of the data. Coefficients inthe model with high weights can be retained as indices that bestdescribe the full spatio-temporal data. When there are a relativelysmall number of temporal samples of the data, sparse regressiontechniques are used to build the model. The model coefficients are usedto perform data processing functions such as indexing, prediction, andclassification.

Magnetic Resonance Imaging (MRI) is an imaging technique to visualizethe internal structure and/or function of a body. MRI provides highercontrast between the different soft tissues of the body than provided bymany other imaging techniques. Consequently, MRI is useful in neurologyand brain imaging. MRI is also useful for imaging other portions of thebody, for example, musculoskeletal, cardiovascular, and for oncological(cancer) imaging. MRI does not use ionizing radiation, but uses apowerful magnetic field to align the nuclear magnetization of, forexample, hydrogen atoms in water in the body. Radio frequency (RF)fields are used to systematically alter the alignment of thismagnetization, causing the hydrogen nuclei to produce a rotatingmagnetic field detectable by the scanner. This signal can be manipulatedby additional magnetic fields to build up enough information toconstruct an image of the body or portions thereof.

Functional magnetic resonance imaging (fMRI) is a type of specializedMRI. fMRI, for example, measures the hemodynamic response (i.e.,response to the dynamic regulation of the blood flow in the brain)related to neural activity in the brain or spinal cord of humans orother animals. Neurons require energy to function. This energy issupplied in the form of glucose and oxygen carried in hemoglobin. Theblood supply of the brain is dynamically regulated to give active neuralassemblies more energy while inactive neural assemblies receive lessenergy. Therefore, changes in blood flow and blood oxygenation in thebrain (collectively known as hemodynamic) are closely linked to neuralactivity. When nerve cells are more active they consume more oxygencarried by hemoglobin in red blood cells from local capillaries. Thelocal hemodynamic response to this oxygen utilization is an increase inblood flow to regions of increased neural activity, occurring after adelay of, for example, 1-5 seconds. This local hemodynamic response mayrises to a peak over, for example, 4-5 seconds before falling back tonear baseline levels, leading to local changes in the relativeconcentration of oxyhemoglobin and deoxyhemoglobin and changes in localcerebral blood volume in addition to this change in local cerebral bloodflow. Therefore, fMRI may, for example, produce images of brain activityby measuring changes in blood oxygenation levels and/or otherhemodynamic responses.

A voxel is a volume element, representing a value, a structure, or athree-dimensional image on a three-dimensional grid. A voxel isanalogous to a pixel, which represents two-dimensional image data.Voxels are frequently used in the visualization and analysis of medicaland scientific data. As with a pixel, a voxel itself typically does notcomprise spatial position or coordinates of the voxel. Rather, spatialposition of a voxel is inferred based on the position of the voxelrelative to other voxels (e.g., the position in the data structure thatmakes up a single volume image). The word voxel is a linguistic blend ofthe words volumetric and pixel.

Embodiments of the invention are useful, for example, in processinglarge amounts of data, such as data produced in conjunction with oranalysis of functional magnetic resonance imaging (fMRI). fMRImeasurements can give rise to large amounts of data, for example,consisting of tens of thousands or hundreds of thousands of voxelsand/or hundreds or thousands of samples, for example, time points orsamples. According to aspects of the invention, fMRI may be used to scanthe brains of subjects, for example, while the brains are receivingstimuli or when brains are diseased or have other states.

According to aspects of the invention, fMRI data (e.g., time-series datacollected at each voxel) may be used to predict the brain, mental orcognitive state of a subject, for example, an emotion (e.g., anger,happiness, sadness, anxiousness, or annoyance); extract patterns of orpredict a mental disease, for example, schizophrenia, depression,Alzheimer's or dementia; discriminate between mental or brain states ofa person, for example brain or mental states associated with a personlooking at a face or at a building, a person listening to a phrase in afirst language or a phrase in a second language, a person performing amental or physical task or different mental or physical tasks, or aperson having one or other emotion; and predicting brain activity givena specific stimulus or specific stimuli (e.g., auditory such as words orsounds, visual such as pictures, or activity of a person such as playinga video-game).

FIG. 1 is a high-level block diagram of a system for modeling andprocessing functional magnetic resonance imaging data according to anexemplary embodiment of the invention. The system 100 comprises aplurality of computational modules including a model builder module 110,a network map generator module 120, an indexing module 130, a predictionmodule 140, and a classification module 150. In addition, the system 100comprises a storage system comprising databases 155, 160, and 170. Thedatabase 160 stores raw spatio-temporal datasets 162 that are acquiredby fMRI time series scans and other secondary time series datasets 164from secondary sources (such as breathing, motion, experimental protocoletc.).

The model builder module 110 processes the raw spatio-temporal data 162and secondary time series data 164 associated with a dataset of a fullbrain scan of a given subject using a multivariate auto-regressiveprocess to generate a full brain vector model of brain activity for asubject brain. In one exemplary embodiment, when there are a relativelysmall number of temporal samples of the data, sparse regressiontechniques are used to build the model. The model builder module 110generates model coefficients 172 for the full brain scan, which arestored in the database 170.

The network map generator module 120 (or network graph generator module)processes the full set of model coefficients 172 of a given dataset andgenerates a reduced representation of the dataset in the form of networkmaps 174 (or network graphs) that are stored in database 170. Asexplained below with reference to FIG. 3, these network maps can begenerated in the form of directed node graphs, where each noderepresents a voxel and where a directed edge (or link) between nodesindicates the degree to which a parent node (voxel) exerts a causalinfluence on a child node (voxel). In general, to generate these nodegraphs, the network map generator module 120 can identify coefficientsin the full model representation having high weights (using one or moretechniques as discussed below), and then retain the coefficients asindices that best describe the full spatio-temporal data to generate thenetwork maps 174 (or network graphs). One exemplary method forgenerating a reduced representation of a full spatio-temporal model ofbrain activity of a subject, which is implemented by the network mapgenerator module 120, will be discussed below with reference to FIG. 4.

The network maps 174 include a summary of the fMRI datasets of variousscans, and are used by the various modules 130, 140, and 150 forindexing, prediction, and classification. For example, the indexingmodule 130 uses the network maps 174 to find fMRI datasets from storedin database 160 which are similar to a given fMRI dataset. The indexingmethods implemented by the indexing module 130 may be used for locatingone or more stored datasets of brains that are functionally similar to atarget brain, or one or more stored datasets brains that perform a sameor similar task while being scanned as the target brain. The indexingmodule 130 can process search requests, whereby a given search request(database query) is used to match the model coefficients of the givenbrain with the model coefficients stored in the database. The datasetsmatching the given model coefficients are returned as the query results.One exemplary method for performing a database search using a vectorrepresentation of a reduced spatio-temporal model of brain activity of asubject, which is implemented by the indexing module 130, will bediscussed in further detail below with reference to FIGS. 5 and 6, forexample.

The prediction module 140 implements methods to determine future fMRIbrain activity in a given voxel using the past information of activityin other voxels and secondary information. The secondary information mayinclude breathing rate, heart rate, subject motion, information from anexperimental protocol, and/or any other information about subjectactivity. The prediction module 140 can also implement methods topredict future secondary information using past values of voxel activityand secondary information. In particular, this includes prediction ofsubject's future brain activity using past data. The autoregressivemodel can also be used by the prediction module 140 to make predictionsof the future activity (in fMRI voxels as well as secondary data). Themodel coefficients can be used to create a variety of maps such as“prediction power”, “prediction accuracy”, “impulse response function,”and other various types of maps. One exemplary method for predictingfuture brain activity using a reduced representation of a fullspatio-temporal model of brain activity of a subject, which may beimplemented by the prediction module 140, will be discussed in furtherdetail below with reference to FIG. 8, for example.

The classification module 150 processes various network maps 174 tocreate classes 155 of datasets that can be used to classify a newdataset into one of various predefined groups. The different groups orclasses 155 may include, e.g., healthy brain vs. diseased brain classes,male vs. female classes, etc. The network maps 174 are used by theclassification module 150 to build a classifier for discriminating amongthe different groups or classes 155. One exemplary method for performingclassification using a vector representation of a reducedspatio-temporal model of brain activity of a subject, which may beimplemented by the classification module 150, will be discussed infurther detail below with reference to FIG. 7, for example.

FIG. 2 is a block diagram of a method for modeling and processingfunctional magnetic resonance imaging data according to an exemplaryembodiment of the invention. In particular, FIG. 2 illustrates anexemplary mode of operation of the system of FIG. 1. Initially, themodel builder module 110 receives a raw spatio-temporal dataset andpossible other secondary information acquired for a given brain scan ofa subject (step 200). The model builder module 110 constructs a fullspatio-temporal model of the brain using the raw data set (step 202). Asdiscussed in further detail below, in one aspect of the invention, amultivariate autoregressive modeling process is used to build a fullbrain model using the raw spatio-temporal dataset.

Once the model is built, the complete set of model parameters of thedata set are stored for further processing (step 204). Using one or moretechniques as discussed below, a set of relevant model parameters fromthe full spatio temporal model are selected to generate a reduced, butaccurate, representation of the full spatio-temporal model (step 206).The selected set of model parameters is then used to build one or morenetwork maps (step 208). In various aspects, the network maps includedirected node graphs that model the functional connectivity of a set ofvoxels within the full brain scan. The network maps represent causalrelations between voxels in the brain and represent dynamics of themodeled process of brain activity. These network maps are used then usedto perform data processing functions (step 210) including indexing,classification and prediction, as discussed herein.

Building a Full-Brain Model

In one aspect of the invention, the process depicted in step 202 of FIG.2 as implemented by the model builder module 110 in FIG. 1 includes aprocess for building a full model of brain activity using anautoregressive model, as will be discussed now in further detail below.In a preferred embodiment of the invention, a stochastic processmodeling the brain activity X (represented as a N dimensional rowvector) is modeled as a linear combination of its past values andindependent, identically distributed (iid) noise. Such representation isalso called a multivariate autoregressive model. Formally, this model isas follows:X(t)=Σ_(τ=1) ^(k) X(t−τ)A(τ)+E(t)  (1)where k is called the model order, A(τ)_(τ=1 . . . k) are the modelparameters in the form of k matrices of size N×N (with coefficientsa_(ij)(t)), E(t) is an N-dimensional row vector of noise with zero meanand a covariance equal to R. For any t₁≠t₂, E(t₁) and E(t₂) areidentically distributed and uncorrelated.

In this model, if a_(ij)(t)>0 for some t, then past values of X_(i)improve the predictability of X_(j) and therefore, X_(i) is said to havecausal influence on X_(j). The parameter t is called the causality lagbetween X_(i) and X_(j).

To infer the causal relationships in the linear simplification, we needto know the model parameters {a_(ij)(t)}. The model parameters may beestimated from a realization of the process X using an fMRI time seriesdataset. Let {x(t)}_(t=1 . . . T) be a realization of the stochasticprocess X and {e(t)}_(t=1 . . . T) be a realization of the iid noise E.This realization must satisfy:x(t)=[x(t−1) . . . x(t−k)][A′(1) . . . A′(k)]′+e(t)  (2)for all tε[k+1, . . . T]. The above set of equations can be written incompact matrix form as follows.

Let Y be a matrix of size (T−k)×N, Z be a matrix of size (T−k)×Nk, W bea matrix of size Nk×N, and

be a matrix of size (T−k)×N obtained by stacking the rows in equation(2) for t=T−k+1 to t=T. Now, Eq. 2 may equivalently be written as:Y=ZW+

  (3)where Y and Z are derived from a realization x of the process X,W=[A′(1) . . . A′(k)]′ contains all the model parameters (a_(ij)(t)) and

is derived from realization e of the noise E.

The maximum likelihood estimate (W_(MLE)) of model parameters can bedetermined by the standard least square solution of Eq. (3), i.e.,

$\begin{matrix}{W_{MLE} = {{\arg\;{\min\limits_{W}{\sum\limits_{j = 1}^{N}{{Y_{j} - {ZW}_{j}}}_{2}^{2}}}} =}} & (4) \\{\underset{a_{ij}{(\tau)}}{\arg\;\min}{\sum\limits_{{j = 1},{t = {k + 1}}}^{N,T}\left\lbrack {{x_{j}(t)} - {\sum\limits_{\tau = 1}^{k}{\sum\limits_{l = 1}^{N}{{x_{l}\left( {t - \tau} \right)}{a_{lj}(\tau)}}}}} \right\rbrack^{2}}} & (5)\end{matrix}$where Y_(j) represents the j^(th) column of Y and W_(j) represents thej^(th) column of W. Eq. (4) has a unique solution only if (3) is notunder-determined, i.e.:(T−k)N≧N ² K

T≧(N+1)k.

In general, for reliable estimates of the model parameters, Eq. (3) mustbe sufficiently over determined, i.e., the number of observations of theprocess X must be significantly larger than the number of modelparameters ((T−k)N>>N²k).

If the model is sparse, i.e., the number of non-zero coefficients in{a_(ij)(τ)} is significantly smaller than the total number ofcoefficients (Nk), then it might be possible to find a reliable solutionto (3) using techniques of sparse regression, as follows.

In particular, consider a multivariate linear regression model of theform Y=ZW where Y is a known n₁×1 response vector, Z is a known n₁×n₂regressor matrix and W is the unknown model vector of size n₂×1 to bedetermined using the response Y and regressor Z. Common methods to solvethis include standard least square regression, ridge regression, andsubset selection methods. For these techniques, it is usually requiredto have n₁>>n₂. However, there is a growing body of work indicating thatif W is sparse, then it may be recovered even if n₂>n₁ using a lassoregression technique, such as described in R. Tibshirani, “RegressionShrinkage and Selection via the Lasso.”, Journal of the RoyalStatistical Society, Serial B, 58(1):267-288, 1996, which isincorporated herein by reference. The lasso regression process solvesthe problem:

$\begin{matrix}{\min\limits_{W}{{Y - {ZW}}}_{2}^{2}} & (6) \\{{s.t.\mspace{14mu}{W}_{1}} \leq t} & (7)\end{matrix}$where ∥.∥₂ ² represents the square of L2 norm and represents the L1 normof the respective vectors. The parameter t is the regression parameterthat is usually chosen after cross-validation.

It can be verified that for any t, there exist a λ such that the program(6, 7) is equivalent to the following optimization problem:

$\begin{matrix}{{\min\limits_{W}{{Y - {ZW}}}_{2}^{2}} + {\lambda{W}_{1}}} & (8)\end{matrix}$

The programs (6, 7) and (8) can be solved efficiently using a techniquecalled least angle regression in time no longer than time required tocarry out the ordinary least square computation. The least angleregression method is known in the art and described in the paper: B.Efron, et al.; Least Angle Regression; the Annals of Statistics 2004;vol. 32(1), pages 407-499, the disclosure of which is incorporatedherein by reference.

The estimation of multivariate autoregressive coefficients in (3) may beviewed as a regression problem where Y is the response variable, Z isthe matrix containing the regressors and W is the model to bedetermined. In this case, the maximum likelihood estimate of (4) becomesthe least square solution to the regression problem. The lassoformulation thus becomes:

$\begin{matrix}{W^{sparse} = {\arg\;{\min\limits_{W}{\sum\limits_{j = 1}^{N}\left\lbrack {{{Y_{j} - {ZW}_{j}}}_{2}^{2} + {\lambda{W_{j}}_{1}}} \right\rbrack}}}} & (9)\end{matrix}$

Note that the coefficients of W_(j) only appear in the j^(th) term ofthe above sum. Therefore, this problem may be decomposed into Nindependent lasso regression problems of size (T−k)×Nk as follows:

$W_{j}^{sparse} = {{\arg\;{\min\limits_{W_{j}}{{Y_{j} - {ZW}_{j}}}_{2}^{2}}} + {\lambda{W_{j}}_{1}}}$

The goodness of fit of the regression is captured using the notion ofwhich predictability (p_(j)) is defined as:p _(j)=1−QΣ _(t=k+1) ^(T) [x _(j)(t)−Σ_(τ=1,l=1) ^(k,N) x ₁(t−τ)a_(lj)(τ)]²where:Q=[Σ _(t=k+1) ^(T)(x _(j)(t))²]⁻¹

It may be verified using the properties of the lasso regression that thepredictability varies from 0 to 1. If the predictability of a voxel is1, the time course of that voxel can be predicted exactly using the pastk values of other voxels. On the other hand, if a voxel has zeropredictability, then the time course of that voxel is orthogonal to(independent of) the shifted time course of all the other voxels.

Selecting Model Parameters to Generate Reduced Model

Once a full-brain multi-variate autoregressive model of brain activityis obtained using techniques described above, the model may be processedin various ways to implement functions such as database indexing,classification and/or prediction of brain activity. In one aspect of theinvention, a full spatio-temporal autoregressive model of brain activitycan be represented in the form of a three-dimensional directed nodegraph comprising a plurality of voxels or nodes say V_(i), where i is anindex, and edges a_(ij)(1) where a_(ij)(1) is a link between voxelsV_(i) and V_(j). The value “1” within the parenthesis, e.g., (1)represents the order of the model. With an order of “1”, a lag of onetime step is used to compute model coefficients. For instance, if a timestep is 2 seconds, then an order of “1” means that a directed link froma parent voxel to a child voxel is computed based on current voxelactivity and past voxel activity 2 second ago. In order words, the modelcoefficients are computed based on current and past voxel activity over1 time step (e.g., 2 seconds), although other model orders such as 2, 3,etc., can be employed.

For instance, FIG. 3 depicts a directed node graph representingfunctional connectivity and causality of one voxel on another voxel,which is derived from modeling functional magnetic resonance image dataaccording to an exemplary embodiment of the invention. In particular,FIG. 3 depicts a portion of a 3-dimensional directed node graph 300showing a pair of voxels V_(i) 310 and V_(j) 320 and a directed edgea_(ij)(1) 330 linking the two voxels V_(i) 310 and V_(j) 320. Thestrength of the link 330 is described by the value of a_(ij)(1). Ifa_(ij)(1) has a value of 0, this means that voxels V_(i) 310 and V_(j)320 do not interact. If a_(ij)(1) has a value of that is greater than 0,this means that voxel V_(i) 310 causes the activity in voxel V_(j) 320to occur. The strength of this causality is greater with higher valuesof a_(ij)(1). In this regard, the dimensionality of the full brain modelmay be reduced by eliminating weak links between voxel in the nodegraph.

For example, FIG. 4 is a flow diagram of a method for generating areduced representation of a full spatio-temporal model of brain activityof a subject, according to an aspect of the invention. Initially, asdiscussed above, a first step is to obtain a full 3-D network node graph(either pre-stored or computed real time) representing voxelconnectivity and causality between voxels in a fMRI scan (step 400). Toreduce dimensionality of the full multi-variate model of brain activity,appropriate features are selected from the multi-variate model. Ingeneral, this process my include comparing the value of the edges(links) between nodes (voxels) to some defined threshold value (step402). Based on the results of the comparison, weak links between voxelscan be eliminated from the network node graph (step 404). A reducednetwork node graph can then be generated (step 406) including a set ofthe voxels having strong links, i.e., edge values that meet or exceedthe predefined threshold.

The dimensionality of a full network node graph can be reduced (viathresholding) in one of many ways. For instance, a suitable thresholdvalue, T, can be used to compare to the value of edges and select thoseedges that may be considered significant. For example, if T is set to avalue of 0.5, then edges a_(ij)(1) can be selected where the value ofa_(ij)(1)>0.5. The effect of this is to eliminate weak links in the fullnetwork node graph. Another method is to first calculate the predictionpower value of a given voxel, as defined above, and then threshold theprediction power value. This can result in a set of voxels, or nodes inthe graph, which are selected to have high prediction power.

Indexing, Classification, Prediction

Once a reduced model representation is obtained, the model may beprocessed in various ways to implement functions such as databaseindexing, classification and/or prediction of brain activity. One way ofprocessing the reduced representation of the full brain model is togenerate a vector representation of the model for use in indexing and/orclassification. In one aspect of the invention, a vector representationis based on the fact that voxels with high prediction power aretypically localized in certain regions of the brain, and correspond tothe specific task that the subject is performing while being scannedwith fMRI. For instance, if the subject performs a motor task liketapping their fingers, then voxels of high prediction power aretypically found in the motor cortex.

FIG. 5 is a flow diagram of a method for generating a vectorrepresentation of a reduced spatio-temporal model of brain activity of asubject for use in database searching and classification, according toan aspect of the invention. An initial step in automatically labeling anfMRI scan is to label each voxel with a spatial index and predictionvalue (step 500). In particular, in one aspect of the invention, thevoxels may be labeled with a 2-tuples (Si, Pi) where Si is the spatiallocation of the ith voxel, and Pi is the prediction power of the voxel.The spatial location Si information can be derived in many ways. Forinstance, the spatial location Si could refer to the specific Brodmannarea that the voxel V_(i) belongs to. Alternately, the spatial locationSi can be an anatomical atlas label, such as that provided by theMontreal Neurological Institute (MNI template).

The next step is to generate a vector representation of the fMRI datasetbased on the set of n-tuples (Si/Pi) associated with the voxels in thatdataset (step 502). More specifically, assume that an fMRI scan givesrise to a set T of 2-tuples, (Si, Pi). One method of generating a vectorrepresentation of T is to flatten the set of 2-tuples in T according tosome pre-defined order, say by concatenating voxels in Brodmann area 1,followed by Brodmann area 2 and so on. This defines a one-dimensionalvector where each location can be mapped back to its Brodmann area, andthe value at each location is the prediction power at that voxel. Thevector representation this then stored in a database and may besubsequently used for database searching and/or classification (step504).

FIG. 6 is a flow diagram of a method for performing a database searchusing a vector representation of a reduced spatio-temporal model ofbrain activity of a subject, according to an aspect of the invention. Aninitial step is to obtain a vector representation of a target scan (step600) using a method as discussed above with reference to FIG. 5. Assumea vector representation is denoted U1 for a dataset T1 of the targetscan. Assume further that a plurality of previous scans are stored in adatabase, each in the form of a vector representation U as discussedabove. Assume further that T2 denotes a stored dataset that is formed bylabeling the voxels in the stored dataset as a set of 2-tuples (Si/Pi)to generate a vector representation U2.

To determine whether T2 (and other stored scan datasets) is similar toT1, a distance measure can be implemented to compare the vectorrepresentation U1 of the target scan dataset T1 with the vectorrepresentation U2 of the stored scan dataset T2 (step 602). There aremany methods that may be implemented to define a distance measure. Inone aspect of the invention, the distance between vector representationsU1 and U2 can be a Euclidean distance D measure, defined as the sum ofthe squares of the differences between the components of U1 and U2, i.e.Σ(U1 _(i)−U2 _(i))². This distance measure allows us to compare two fMRIscan datasets T1 and T2. If the determined distance D between U1 and U2is small, it is determined that the scans T1 and T2 are similar, andmost likely constitute similar experimental protocols. This distancemeasure can be used as a database search mechanism, whereby the databasesearch engine can retrieve one or more stored scans that are closest indistance to the target fMRI scan dataset (step 604).

In other aspects of the invention, a vector representation U for a scandataset can be used for classification. For example, FIG. 7 is a flowdiagram of a method for performing classification using a vectorrepresentation of a reduced spatio-temporal model of brain activity of asubject, according to an aspect of the invention. In particular, FIG. 7illustrates a method for using a vector representation U of a scandataset T to train a classifier to identify different instances of thevector U. An initial step is to obtain a vector representation U foreach of one or more scan datasets (step 700). Next, each vectorrepresentation U is associated with a class label L, which may describea characteristic (e.g., mental state) of the associated individuals(step 702). For instance, the class label L may indicate whether anindividual is diseased or healthy. Thus, we obtain labeled 2-tuples,(U1, L1), (U2, L2), . . . (Un, Ln), where there are n samples of fMRIscan datasets. The labeled 2-tuples, (U1, L1), (U2, L2), . . . (Un, Ln)are then used train a suitable classifier to determine a decisionboundary that can separate the different classes according to the classlabel L (step 704).

For instance, we can distinguish instances of class A (e.g., subjects inclass A) from instances of class B (e.g., subjects in class B). One suchclassifier is the support vector machine, which is described in thebook: The Nature of Statistical Learning Theory by Vladimir Vapnik,Springer-Verlag publisher, 1995, ISBN 0-387-98780-0, the disclosure ofwhich is incorporated herein by reference. It is to be appreciated thatother suitable classification methods may be implemented in accordancewith aspects of the invention, the details of which are well understoodto those of ordinary skill in the art.

In other aspects of the invention, an auto-regressive model can be usedto predict the future time course of activities in different brainvoxels. FIG. 8 is a flow diagram of a method for predicting brainactivity using a reduced representation of a full spatio-temporal modelof brain activity of a subject, according to an aspect of the invention.As an initial step, a full auto regressive model of brain activity isobtained for a target subject performing a given task (step 800) andreduced in form using the methods discussed above with reference to FIG.4. Subsequently, this reduced model representation can be used topredict future brain activity for the target subject while performingthe same task without having to obtain a full scan. In one aspect, whenperforming a prediction process, the previously generated model isretrieved from a data storage, and the model is initialized with thetarget subject's current brain activity at some arbitrary time instant,T₀, while performing the same task (step 802).

More specifically, with this process, the target subject will performthe same task that he/she previously performed when scan data waspreviously collected to generate the full autoregressive model of brainactivity for the given task. For this process, while the subject isperforming the same task, a brain scan is performed to collect someinitial brain activity data that is used to initialize the model. Themodel coefficients are then used to predict the future brain activity ofthe target subject at times T>T₀ (step 804). More specifically, themodel coefficients are used to predict how the activity in the brain ofthe target subject will evolve over time for the given task withouthaving to perform a full brain scan. The model is specific to theindividual subject and the activity being performed. So when the modelis initialized with a current estimate of voxel activity, at time t−T₀,the model coefficients can be used to predict future brain activity inregions of the subject's brain at subsequent times, T₁, T₂, T₃, . . . ,where an accurate prediction of brain activity may be obtain for severalsecond or tens of seconds after T₀, depending on the model.

The predicted brain activity can then be machine interpreted to providereal-time biofeedback to the target subject being scanned based on thepredicted future brain activity (step 806). This form of biofeedback canbe used for therapeutic purposes. For instance, an individual may have astuttering problem. During an initial scan, the subject may perform agiven activity that involves reading a sentence with instructions toverbally repeat the sentence aloud. During this task, a full brain scanmodel is generated, which may capture a sequence of brain activity thatindicates a particular dysfunction in the brain processing which leadsto stuttering. Thereafter, the auto-regressive model for the givensubject can be used to interpret the future time course of brainactivity as follows. The target subject is subsequently scanned whilebeing shown the same specific sentence as previously shown for the fullscan, and asked to read it aloud. The brain activity right after thesentence is recognized can constitute the initial condition that isapplied to the previously generated model. Thereafter, the model is usedto predict whether the motor cortex becomes active in the future. If themodel predicts that the motor cortex will not become active and theperform is likely to stutter, corrective biofeedback can be provided tothe subject, such as a light or a sound. This allows the person to takeappropriate corrective action, such as controlling their breathing orarticulation, to avoid or mitigate stuttering.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method, or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of fauns, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wire line, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present invention are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

It will also be noted that each block of the block diagrams and/orflowchart illustration, and combinations of blocks in the block diagramsand/or flowchart illustration, can be implemented by special purposehardware-based systems that perform the specified functions or acts, orcombinations of special purpose hardware and computer instructions.

Accordingly, aspects of the invention, for example as depicted in FIGS.1-8, can also include, as described herein, providing a system, whereinthe system 100 of FIG. 1 includes distinct software modules. By way ofexample only, the software modules may include the model building module110, the network map generator module 120, the indexing module 130, theprediction module 140, and the classification module 150. The modulesmay be adapted, for example, to perform the steps of those methods asillustrated in FIGS. 2-8.

One or more embodiments can make use of software running on a generalpurpose computer or workstation. With reference to FIG. 9, such system900 employs, for example, a processor 902, a memory 904, and aninput/output interface formed, for example, by a display 906 and akeyboard 908. The term “processor” as used herein is intended to includeany processing device, such as, for example, one that includes a CPU(central processing unit) and/or other forms of processing circuitry.Further, the term “processor” may refer to more than one individualprocessor. The term “memory” is intended to include memory associatedwith a processor or CPU, such as, for example, RAM (random accessmemory), ROM (read only memory), a fixed memory device (for example,hard drive), a removable memory device (for example, diskette), a flashmemory and the like. In addition, the phrase “input/output interface” asused herein, is intended to include, for example, one or more mechanismsfor inputting data to the processing unit (for example, keyboard ormouse), and one or more mechanisms for providing results associated withthe processing unit (for example, display or printer). The processor902, memory 904, and input/output interface such as display 906 andkeyboard 908 can be interconnected, for example, via bus 910 as part ofa data processing unit 912. Suitable interconnections, for example viabus 910, can also be provided to a network interface 914, such as anetwork card, which can be provided to interface with a computernetwork, and to a media interface 916, such as a diskette or CD-ROMdrive, which can be provided to interface with media 918.

A data processing system suitable for storing and/or executing programcode can include at least one processor 902 coupled directly orindirectly to memory elements 904 through a system bus 910. The memoryelements can include local memory employed during actual execution ofthe program code, bulk storage, and cache memories which providetemporary storage of at least some program code in order to reduce thenumber of times code must be retrieved from bulk storage duringexecution.

Input/output or I/O devices (including but not limited to keyboard 908,display 906, pointing device, and the like) can be coupled to the systemeither directly (such as via bus 910) or through intervening I/Ocontrollers (omitted for clarity).

Network adapters such as network interface 914 may also be coupled tothe system to enable the data processing system to become coupled toother data processing systems or remote printers or storage devicesthrough intervening private or public networks. Modems, cable modem andEthernet cards are just a few of the currently available types ofnetwork adapters.

As used herein, including the claims, a “server” includes a physicaldata processing system (for example, system 912 as shown in FIG. 9)running a server program. It will be understood that such a physicalserver may or may not include a display and keyboard.

It will be appreciated and should be understood that the exemplaryembodiments of the invention described above can be implemented in anumber of different fashions. Given the teachings of the inventionprovided herein, one of ordinary skill in the related art will be ableto contemplate other implementations of the invention. Indeed, althoughillustrative embodiments of the present invention have been describedherein with reference to the accompanying drawings, it is to beunderstood that the invention is not limited to those preciseembodiments, and that various other changes and modifications may bemade by one skilled in the art without departing from the scope orspirit of the invention.

The invention claimed is:
 1. A method to perform an image dataprocessing operation, comprising: obtaining a raw spatio-temporaldataset acquired from scanning a brain of a subject performing a giventask; constructing a full spatio-temporal model using the rawspatio-temporal dataset, wherein the full spatio-temporal modelrepresents brain activity that occurs in all regions of the subject'sbrain in response to the subject performing the given task; selectingmodel parameters from the full spatio-temporal model which meet orexceed a threshold parameter that defines a level of causal relationbetween voxels in the acquired dataset; generating a reduced modelrepresentation of the full spatio-temporal model using the selectedmodel parameters; generating a vector representing the reduced model,wherein generating a vector representing the reduced model comprises:labeling each voxel with a spatial index Si and a prediction value Pi;concatenating voxels having a common spatial index Si; and generating avector comprising sets of concatenated voxels and a correspondingprediction value for each set of concatenated voxels; and using thevector to perform an image data processing operation, wherein the methodis performed by a computer.
 2. The method of claim 1, wherein the rawspatio-temporal data comprises functional magnetic resonance imagingdata.
 3. The method of claim 1, wherein constructing a fullspatio-temporal model comprises using a multivariate auto-regressivemodel to generate a model of the spatio-temporal dataset.
 4. The methodof claim 1, wherein selecting model parameters from the fullspatio-temporal model which meet or exceed a threshold parameter thatdefines a level of causal relation between voxels in the acquireddataset, comprises: obtaining a value of a causal link between a pair ofvoxels; comparing the value of the causal link to the thresholdparameter; selecting all causal links between pairs of voxels that meetor exceed the threshold parameter; and eliminating all causal linksbetween pairs of voxels that do not meet or exceed the thresholdparameter.
 5. The method of claim 4, wherein the threshold parameter isbased on a prediction power threshold value that specifies a thresholdfor which brain activity in a given voxel predicts brain activity inanother voxel.
 6. The method of claim 1, wherein generating a reducedmodel representation of the full spatio-temporal model using theselected model parameters comprises generating a three-dimensionaldirected node graph having nodes and directed edges connecting thenodes, wherein nodes in the directed node graph represent voxels andwherein a directed edge in the directed node graph represents a causalrelation between two nodes connected by the directed edge.
 7. The methodof claim 1, wherein using the vector to perform an image data processingoperation comprises performing a database indexing operation, whereinperforming a database indexing operation comprises: generating a vectorfor a target brain scan; using a distance measure to compare the vectorfor the target brain scan with one or more vectors representingcorresponding brain scans stored in a database; and retrieving one ormore stored brain scans that are close in distance to the target brainscan based on a distance measure between vectors representing saidtarget and stored brain scans.
 8. The method of claim 1, wherein usingthe vector to perform an image data processing operation comprisesperforming a classification operation, wherein performing aclassification operation comprises: generating a vector for each of aplurality of different brain scans; applying a class label to each ofthe vectors indicating a characteristic of the associated scans; andtraining a classification system using labeled vectors of the brainscans.
 9. A computer program product to perform an image data processingoperation, the computer program product comprising a non-transitorycomputer readable storage medium having computer readable program codestored thereon, the computer readable program code comprising computerreadable program code which when executed by a computer, performs themethod of claim
 1. 10. An apparatus to perform an image data processingoperation, the apparatus comprising: a memory storing programinstructions; and a processor coupled to the memory, operative toprocess the stored program instructions to: obtain a raw spatio-temporaldataset acquired from scanning a brain of a subject performing a giventask; construct a full spatio-temporal model using the rawspatio-temporal dataset, wherein the full spatio-temporal modelrepresents brain activity that occurs in all regions of the subject'sbrain in response to the subject performing the given task; select modelparameters from the full spatio-temporal model which meet or exceed athreshold parameter that defines a level of causal relation betweenvoxels in the acquired dataset; generate a reduced model representationof the full spatio-temporal model using the selected model parameters;and store the reduced model for use in performing an image dataprocessing operation, wherein the image data processing operationincludes one of image indexing, classification, and predicting futurebrain activity, wherein the processor is further operative to processthe stored program instructions to: generate a vector representing thereduced model, wherein generating a vector comprises labeling each voxelwith a spatial index Si and a prediction value Pi, concatenating voxelshaving a common spatial index Si, and generating a vector comprisingsets of concatenated voxels and a corresponding prediction value foreach set of concatenated voxels; and use the vector to perform imageindexing and image classification.
 11. The apparatus of claim 10,wherein the processor is operative to process the stored programinstructions to predict future activity by: obtaining the stored reducedmodel; initializing the reduced model with current brain activity dataderived from a scan of the subject's brain while the subject isperforming the same given task; and predicting future brain activity ofthe subject using the reduced model based on the current brain activitydata initializing said reduced model.
 12. The apparatus of claim 10,wherein the processor is operative to process the stored programinstructions to use the vector to perform an image indexing operation bygenerating a vector for a target brain scan, using a distance measure tocompare the vector for the target brain scan with one or more vectorsrepresenting corresponding brain scans stored in a database, andretrieving one or more stored brain scans that are close in distance tothe target brain scan based on a distance measure between vectorsrepresenting said target and stored brain scans.
 13. The apparatus ofclaim 10, wherein the processor is operative to process the storedprogram instructions to use the vector to perform an imageclassification operation by generating a vector for each of a pluralityof different brain scans, applying a class label to each of the vectorsindicating a characteristic of the associated scans, and training aclassification system using labeled vectors of the brain scans.